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ABSTRACT 

The TITA (Totally Interactive Testing and Analysis) 
System algorithm for the repetitive construction of domain-referenced 
tests utilizes a compact data bank, is highly portable, is useful in 
any discipline, requires modest computer hardware, and does not 
present a security problem. Clusters of related keyphrases, statement 
phrases, and distractors form rainipools from which the computer 
generates items for a domain-referenced unit of instruction. Test 
items can take the form of multiple-choice, true-false, matching, and 
fill^-in questions. A random number generator produces data for test 
items requiring numerical solutions, and the correct answer is 
computed from a coded formula so computational subroutines are not 
required for each test item. This component of computer managed 
instruction allows the instructor to key related items in the data 
minipool to learning resources and to code the resources themselves 
for inclusion in the data bank. Use of this system for elementary, 
secondary, or undergraduate courses can facilitate instructional 
management and result in positive effects on student morale. (CH) 
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INTRODUCTION 

Testing, perhaps more than any other single educational activity, 
traditionally has been viewed as a "necessary evil". In general, students 
view it with anxiety while many teachers, at all levels of instruction, 
consider writing and grading exa-ninations a tedious clerical task that 
takes avs-ay valuable tUne which otherwise could be used for more rewarding 
activities, e.g., individual student counsellirig. That testing is necessary 
in education is at best a mute question since it provides one of the most 
straightforvrard means of quantifying a student's achievement. That it is, at 
the same time, viewed irtth undeserved trauma bespeaks its own shortcomings. 

Most students • anxieties over an examination arise from fear that they 
"do not know what will be asked", as though the test is a guessing game where 
one's cliances of success depend as much on luck as one's mastery of the 
subject matter. Removing the "guessing" out of examinations is an jjnportant 
feature of domain-referenced (as opposed to norm-referenced) testing where 
test items are confined to predefined behavioral objectives and cover only 
the degrees of rrastery, e.g., knowledge, comprehension, etc. in Bloom's 
taxonomy!, expected of the student. It is no accident that students in 
PSI (personalized system of instruction) courses possess a healthy attitude 
toward examinatioas; the mastery model requires that a student be given only 
donain-referenced examinations for (in principle) as often and as long as it 
takes for him to gain ccnplete mastery of a course unit. Failed examinations 
carry no penalty except for lost time. Unfortunately, an instructional 
strategy, such as PSI, that attempts to individualize instruction entails 
a considerable amount of time and effort for course m.anagement. It is 
precisely vd.thin the province of instructional management that the computer 
is maldng its most dramatic impact uvon education in recent year-s. Central 
to any successful computer-rrar.a;;ed instruction CCT-lIl system is the capability 
to generate repeatable domain-referenced examinations. V-Tiile computerized 
test construction has becane increasingly popular In recent years^ man;/ of 
the techniques in U3e today are either modest modifications of the simple 
item-retrieval syr-tem or else involve the construction of a massive data 
bank and/or a complex set of highly speciallze-d subroutines. 
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At Washington Tech our (ML strategy centers upon a system called 
Totally Iterative Testing and ^jialysis (TITA) \vhich incorporates a versatile 
algorithm for the conputer generation of repeatable examinatior^ . The 
algorithm is superior to many others in current use. The algorithm's 
attractiveness is fourfold: (l) it utilizes an extremely conpact data bank; 
(2) it is highly portable; it is useful for any discipline since it does not 
depend heavily on the conpact data bank; (3) it requires modest computer 
memory or hardware, and (/I) it does not present a security problem of any 
kirxi. Since the procedure was first reported3 we have made significant 
strides toward increasing its flexibility. 

IMS paper describes the Improved technique for generating repeatable 
examinations and provides an overview of our ail strategy where the test 
generation procedure plays a key role. 



TEST GEIERATION ALGORmM: GROSS FEATURES 

The versatility of the algorithm essentially results from the fact that 
the data banic does not contain the full test questions but only their parts 
so that the computer has the freedom to assemble these parts to produce a 
logical test question in a completely random fashion. In order to see this 
more clearly, consider a multiple choice question as essentially consisting 
of two parts, namely, the question stem and the list of choices. The latter 
Includes the correct answer and several distractors. Now, the question stem 
itself may be divided into two parts: (1) the keyword or keyphrase, and 
(2) the "statementphrase". Hhus, in the simple test question 

AmW IS THE SEAT OF GOVERi^TT OF 
(A) ALABA'^ (B) mvl YORK (C) mi HAMPSxHIRE . (D) MAHffi 

the question stem contains the key^-;ord, "Albany", and the statementphrase, 
"is tlie seat of governnient of" . 

The data bank consists of clusters of related keyphrases, statc-jr.entphrases 
and distractors, which, when taken together, constitute a data "mini-pcol". 
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For exarole, the test iten ju*t cited could be fonnulated from a mini-pool such as 



KEYPKRASE POOL 
Albany 
Montgcnery 
Augusta 
Concord 
Columbia 



STATEJENTPHRASE POOL 
is a city located in 
is the capital of 
is the seat of government of 



DISTRACTOR POOL 
New York 
Alabama 
New Hampshire 
Maine 

North Carolina 
South Carolina 



It is clear that a multiple-choice question may be formulated from a 
given mini-pool by selecting one member of the keyphrase poox, joining it with 
one member of the statementphrase pool and selecting any three (or four) 
distractors in addition to the correct answer to make up the list of choices. 
The order In vMch the distractors and correct answer appeal^ in the list of 
choices is, of course, always randan. Occasionally, depending on the cui^ent 
output of the randan number generator, an additional choice, e.g., 
"(E) NO.\E OP THESE", may be included. By design, the probability of this 
being the correct answer (in which case, the other four choices must all be 
distractors) is extremely small. 

There is no llnit on the number of pool members; Indeed, greater 
variability of generated test items results from the inclusion of more 
ma-nbers. In addition, each pool member may be a single word or a group of 
words totallL-^ several printed lines so that one is not limited to the 
formulation of short test itejns. 

ITie mini-pool cited above is an exa-nple of a "sJjnple" mini-pool in that 
one correct answer is associated vdth a given keyphrase regardless of which 
staterientphrase is used. This is because the members of the statementphrase 
pool are analogous to each other. It is insti'uctive to note that our 
technio;je is not li-nited to the creation of sinple mini-pools . The 
repeatability of the computer generated test and the compactness of the 
data kark arise, ir. fact, fran the use of "car.plex" mini-pools where the 
correct answer assccia-&a with a given keyphrase depends upon which 
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st«,..e , ^^^^ ^^^^^^ 

A solution of 0.001 M « rr „ 

H2SO2, has a pF of about 2.7 

A O.OBf solution of ha<? « nnw ^ 11.3 

sulfuric acid nas a pOH of approxijTately ^ 

3 • 0 

AO, Hi KOH solution 2.0 
A 0.001 M HCl solution 11.0 

12.0 
1.7 

12.3 

approaches. oistractor pool as in more traditional 

TOUE/PAI^E, Ca-IPLEiTON AJD MATCHING TYPE l^ST m-IS 

.uestr:::: r™^: r ^^"^^r ^^^^^ ^-^^ ^ 

cui ue recast both as a true/false or fin in kt , 
In the fomer case th^ . l lil-in-the-blanl< question. 

conoarativelv ^^tpii o<-„^ ^ is 
.c-aoxveiy small, students are reaulrpd o^^J,^v, 

(excep. .or constructs, t.e yJTL ^ rl^^V^ " 

words, a ccroletion test oi " ^ ^"^ °^her 

either Place the^:L:~^ ^ "'T^''" 

the correct answer (or dist>^a'torTr ° '° ^'^^ ""'^^^^ 

^or dist. actor) is supposed to go. Generally, there is 
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less or no ainbiguity in the latter. For exan¥)le, in the question intended 
to read as 

GROUP VLIIA ELEJOTTS ARE CALLED NOBLE OR BEFff GASES 

the tTO possibilities are: (1) are called noble or inert gases, and 

(2) Group VIIIA elements are called . The first case is ambiguous 

in that its blank iray be filled correctly by any one of the follovdng: 
(a) Ke and Ne (or various combinations of Group VIIIA elements); (b) The 
chemically un2?eactive group of elenents; (c) Elements with zero valence. 

The division of a test item into a keyphrase, statementphrase and 
distractor portion readily permits its being included in matchlng-type sets 
of questions. To acconplish this, an entire mini-pool is gathered and the 
members of the stater.entphrase pool are discarded. The test items then involve 
matching the set of keyphrases with the corresponding set of distractors. It 
should be noted that menbers of complex mini-pools are not normally suited 
for fomulation as matching-type test items. 

It is clear tl\at our technique allows the conputer to present a given 
test item in any one of several different types using exactly the same 
compact data bank. 

THE DATA BAfJK 

The data bank is arbitrarily divided into sections corresponding to 
each unit of a course associated v/ith specific coarse objectives, viz , the 
domain. Test itens in the bank are either ''root" or "branch". Each item 
has a unique question number and a keyphrase. If the item is a root it^, 
it also contains the statementphrase pool arid the distractor pool. A branch 
iteti is associated v:ith a specific root itec^ and appears in the data banl< 
vdthout either a statementphrase pool or distractor pool. Thus, the branch 
itera is rjch shorter and stapler than a root iton and needs only to point to 
its associated root item in order to obtain the required pools. SLnce there 
is no llrdt on the nurier of branch items associated v.lth a given root item. 
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the cocroactness of the data bafJk is evident. 



NUMERICAL PROBLEMS 



The test generator is now also capable of fonnulating test items 
requiring nirierical solutions where the required data are produced by the 
randcn nun±)er generator at execution time. This feature enables one to 
construct truly repeatable examinations while still preserving the compactness 
of the data bank ard the portability of the onLire package. The correct 
answer is ccrxuted by the rnachlne fron a coded formula so that one does not 
need to write a separate subroutine for each test item. The data bank entry 
for this type of test item is different fron the others which are formulated 
from data nlni-pools. Here, the question is written in full, except that the 
pound sign (^) appears v;here a number is to be randomly generated. For example, 
a test itCT may be 



ff MOLES 0? A:I ideal gas at 0 m ATM OCCUPY A VOLUME (LTIERS) OP 

In the bank this particular question is followed by six numbers with the 
correct number of significant digits, which define the upper and lov;er bounds 
for the three data. These are, in turn, followed by a coded formula which the 
program uses to compute the correct ansv/er, e.g., 1*0.082* (2+273. l)/3, 
where fixed point numbers refer to the internally generated data and floating 
point numbers are numeric constants which are part of the formula. Test 
items such as this still may be cast as multiple choice, true/false or 
fill-Ln-the-blarJ<. 



The fact that test itens In the bank are all domain-referenced, in addition 
to the clustering of related Items in data mlni--pools, greatly facilitate the 
management of iastruction. Each mini--pool is keyed to a set of learnlrig 
resO'Urces wriich a student nay consult for m.ore information or in-depth study. 
The learnir^ r eschar ces key ray be included in the data bank, althoaf:h we use 
a separate file for this purpose, The learnirig resources are coded vrith 
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a five-digit number where the first dl^it identifies the type of learning 
resource, e.g., film, cassette, CAI molule, textbook, and the remaining 
digits provide specific infonration about the resource, e.g., pages of a 
reference book, film number, etc. 

A student, learning for mastery, takes a computer-generated test and 
hands his cai?)leted paper to his instructor or proctor who then feeds the 
student answer sheet to an optical reader and grader program. The grader 
progra-n, not only scores the test, but also updates the student's course 
history, performs the required statistics, such as item analysis, and uses 
the learning resources file to prescribe a program of study. The cycle may 
be repeated IT necessary since the test generator is capable of providing a 
set of unique but equi'/alent exaninations. 

\-Jhat a Cr4I of this sort does for the student morale is inestimable. 
Its success as an instructional strategy is guaranteed by at least two 
factors: (1) the impairment of student learning brought on by the usual 
delay between examination tlm.e and the time the graded paper is returned 
to the student is avoided; (2) the unique examination each student gets serves 
as a valuable study aid. It eliminates student anxiety over examinations, 
and also accelerates student learning by encouraging students to discuss 
course raterials a-nong themselves. 
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